SilverStripe CMS browser testing overview 


Introduction 

In March 2011, we tested the content-authoring interface within SilverStripe 
CMS against various major browsers, in order to provide recommendations to 
our users, and to identify opportunities for improvement. This document reports 
on our findings. 


The browsers tested were Internet Explorer (IE) 7, IE8, IE9, and Firefox 4 for 
both Windows and Mac. We chose not to use IE6 as it is no longer supported by 
Microsoft, and now has only a 5% share of the global browser market. 


Similarly, as testing was done primarily from a content author’s perspective, we 
did not test in Chrome or Safari, as our statistics do not show a significant 
number of users in this market. At some stage in the future we may incorporate 
these browsers, and we would welcome community involvement in their testing. 


All tests were done using SilverStripe CMS’s minimum browser resolution of 
1024x768 px. We did the testing with our screen set to this resolution. To help 
identify problems found only in cramped screen resolutions. The testing was 
done on both a simple ‘out of the box’ installation as well as complex website to 
mirror a sophisticated production website. 


We have included our testing methodology and individual results as appendices 
for your information. 


Internet Explorer 7: 

This browser is the minimum version of Internet Explorer (IE) required to run 
SilverStripe CMS. All the functionality of SilverStripe CMS that we tested worked 
in this browser, however some tasks were slow. If you are currently using IE7 to 
access SilverStripe, we recommend that you upgrade to IE9, as this will offer a 
significant speed improvement when carrying out some common tasks within 
SilverStripe CMS. IE9 also contains security and other incremental 
improvements that improve your overall web browsing experience. The IE7 
browser version we used to test was 7.00.5730.13. 


Internet Explorer 8: 

All tested functionality within SilverStripe CMS works well, and functionality 
runs more quickly than in IE7. The IE8 browser version we used to test was 
8.0.6001.18702. 


Internet Explorer 9 

As of April 2011, IE9 is Microsoft's latest browser, and the preferred Internet 
Explorer version for running SilverStripe. The IE9 browser version we used to 
test was 9.0.8112.16421. 


Firefox 4.0 (Windows): 
All SilverStripe CMS functionality worked well. Performance was similar, and in 


several cases better, than IE8. Firefox 4 has better support of web standards 
(HTML, CSS, JavaScript) than Internet Explorer, and this offers users a small level 
of usability improvements. 

Because Firefox provides the best experience, it is therefore our recommended 
browser for content authors using Microsoft Windows. 


Firefox 4.0 (Mac): 

All SilverStripe CMS functionality worked well. Performance was similar, but 
occasionally slower, than Firefox Windows. There are known issues with using 
Safari with SilverStripe CMS, therefore Firefox 4.0 is our recommended browser 
for Mac OS X content authors. 


Performance tests 


Note about performance times shown in benchmarks 

The time you experience it takes to perform tasks with SilverStripe CMS are 
likely to be faster or slower than those displayed in the tables in this document. 
Hardware, software, your web server and PHP configuration, and the number of 
applications you have running will make a major impact on the benchmarks. 


However, the numbers serve to identify useful comparisons between the 
browsers, providing objective information on which browsers are quicker, and 
by what factor. It also helps us understand if there are features in SilverStripe 
CMS which could be rebuilt in order to deliver a faster performance. 


1. Logging into the CMS with different installations 






IE7 





IE8 








IE9 





Firefox4 PC 2.34 5.56 











Firefox4 Mac 





The size of the installation seems to have an effect on login speeds. IE8 is 
surprisingly slower than IE7 in this instance. However, as the average user really 
only logs into a site once or twice a day, generally this time does not matter so 
much in the grand scheme of things. 


2. Time from clicking a default page in site tree until you can begin editing on 
different installations 


IE7 





IE8 1.62 6.45 





IE9 








Firefox4 PC 








Firefox4 Mac 














There is a notable increase in time for an advanced installation over all browsers. 
The best browser to use in this case would be Firefox PC, closely followed by IE9. 
Navigating the site tree is a repeated task, and it is important that the time it 
takes is as low as possible to make working with the site stress-free. 


3. Time to create a single page on different installations 


























IE7 

IE8 

IE9 1.31 5.55 
Firefox4 PC 1.11 6 
Firefox4 Mac 09 5.94 








Waiting 17 seconds to create one page is too long, so if you have an advanced 
site, we’d recommend using IE9 or Firefox. 


4: Time to save a single page on different installations 
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IE8 
IE9 
Firefox4 PC 1.58 6.92 
Firefox4 Mac 1.89 9.46 














28 seconds is far too long to wait to save a page. This shows that IE9 and Firefox 
offer significantly better performance for saving pages in SilverStripe CMS. 


5: Time to publish a single page on different installations 
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Although publishing a page in a basic installation is a pretty quick task in all 
browsers, for an advanced installation, IE7 is much slower. This is important, as 
you may often want to make a quick change to a page, and publish it quickly too. 


6. Time to unpublish a single page on different installations 
















IE7 0.56 
IE8 0.56 
IE9 

Firefox4 PC 








Firefox4 Mac 








All browsers perform reasonably quickly. 


Performance conclusions 

For small installations, no browsers had difficulty performing all tasks quickly. 
More complicated sites, however, put more pressure on the installation and 
cause commonly performed sites to slow down. Using a more up-to-date 
browser such as IE9 or Firefox 4 will offer improved performance. It is likely that 
the JavaScript and PHP code in SilverStripe CMS can be optimized. This 
document outlines where we can make performance improvements for future 
releases of SilverStripe CMS. 


Functionality testing 


Functionality tests 

A number of functionality tests were run using IE7, IE8, IE9 and Firefox 4 (PC 
and Mac) to discover if functionality was broken or buggy in those browsers, and 
to come up with recommendations about any areas that could be changed or 
improved. 


Functionality conclusions 


All browsers (IE7, IE8, IE9, FireFox4 on PC and Mac) worked reliably and no 
major problems were discovered. Therefore, users can confidently use any of 
these browsers, with the advantage of using more modern browsers being 
primarily to do with performance and the productivity this provides. 


No new issues arose in Internet Explorer 9 or Firefox 4 that were not already 
apparent in Internet Explorer 8 or Firefox 3 respectively. Therefore, the issues 
that were discovered were already deemed minor and already in our public bug 
tracker at http://open.silverstripe.org and no other action was needed to taken. 





Appendices 


Tester and date 


Testing was done by SilverStripe Ltd employee Michael Andrewartha in March 


2011. 


Assumptions: 


The advanced installation is assumed as the main SilverStripe test case 
The advanced installation is also assumed to be a very complex 
installation 

Not all SilverStripe sites are as complex as the advanced installation so 
the simple installation is used to represent a default installation with a 
couple of modules which are widely used on SilverStripe sites 

Windows is assumed to be the most popular operating system for 
SilverStripe CMS content authoring. 

1024x768 is assumed as a broad browser resolution so will be considered 
as the default browser resolution and all browsers will be tested using 
this resolution. Any higher and you limit the bulk of users using 
SilverStripe. 

Limited resources meant only one browser could be tested on Mac, and 
we chose Firefox in order to compare it to Firefox on the PC. 

Only the SilverStripe CMS back end will be tested . All user tests are done 
on the same test server . Two sites will be tested concurrently with the 
same test 

All the performance user tests are designed to be run on both sites 

The tests will be tailored to a user’s needs, hence outlining each use case 
followed by the outcomes of the use case on each browser 

Different test programs will be used for IE (AOL Pagetest) and for Firefox 
(Firebug), however it is assumed that as they are testing the same thing, 
so the results will be similar. 


Basic installation of SilverStripe 2.4: 


Default installation of SilverStripe, basic modules installation, revisions 
set similar to advanced installation for accurate testing, incorporating 25 
pages. 
Modules: 
o CMS 
o Sapphire 
o Blog 
o Userforms 


Advanced installation of SilverStripe 2.4 


A website created for a client by the SilverStripe company, implemented 
on the SilverStripe test server, revisions set, incorporating 2238 pages. 
Modules: 

o CMS 

o Sapphire 

o blog 


calendar 
cmsworkflow 
contentreview 
events 

forum 
genericdataadmin 
iframe 

linkchecker 
legacydatetimefields 
multiform 
payment_dpshosted 
quicklinkscollections 
recaptcha 
spamprotection 
Sphinx 
staticimporter 
subsites 

userforms 
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Browsers: 
e JE7 (7.00.5730.13) 
e IE8 (8.0.6001.18702) 
e JE9 (9.0.08112.16421) 
e Firefox 4.0 (PC) 
e Firefox 4.0 (Mac) 


Environment: 
e Windows testing is done in a VirtualBox environment (v3.2.6 r63112) 
e Mac testing is done using Leopard (v10.5.8) 


Performance test programs used: 
e AOL Pagetest: IE7, IE8 , and IE9 performance 
e Firebug: Firefox performance 


Performance user cases list 
These tests are used to compare a basic installation with an advanced 
installation: 

e Time for logging into the CMS 

e Time from clicking a default page in site tree until you can begin editing - 

Time to create a single page 

e Time to save a single page 

e Time to publish a single page 

e Time to unpublish a single page 


Functionality user cases list 
These tests are used to test browser issues. Functionality issues in the 
performance tests will also be noted but some may not apply to the basic site: 


Search site tree for a title 

Search site tree page type 

Publish multiple pages (batch actions) -Delete multiple pages (batch 
actions) 

Show deleted pages 

Show changed pages 

Drag and drop a page in site tree 

Right click site tree and duplicate page 

Right click site tree and sort sub pages 

Page version history - View 

Page version history - Compare 

Page version history - include unpublished pages 
Site reports - pages edited in the last 2 weeks 
Content - Add an image 

Content - Add a link 

Content - Paste as plain text 

Content - Paste from word 

Behaviour - change page type 

Behaviour - Change page location 

Access - set access conditions 

Attach document to page 

Create a new folder 

Upload a file via files and images 

Drag files to different folders 


